Skip to content

feat: unify Noema, OpenCode, and Strix free-first fallbacks - #760

Draft
seonghobae wants to merge 12 commits into
mainfrom
feat/unify-free-first-llm-fallbacks
Draft

feat: unify Noema, OpenCode, and Strix free-first fallbacks#760
seonghobae wants to merge 12 commits into
mainfrom
feat/unify-free-first-llm-fallbacks

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Unifies the model-selection policy used by Noema, OpenCode Agent, and Strix without replacing their existing provider transports, reviewer identities, credential scopes, result validators, or security gates.

  • imports the exact integrated contextual-orchestrator fallback-policy source through a receipt-verified vendored module;
  • validates one strict, versioned organization model manifest;
  • deterministically exhausts every eligible free candidate before any paid candidate;
  • prefers free-to-free fallback before paid escalation;
  • filters by repository visibility, required capability, and configured credential name;
  • rejects duplicate targets, unknown fields, unsafe identifiers, source drift, symlinks, duplicate JSON keys, and empty eligible pools;
  • retains the unchanged Noema, OpenCode, and Strix execution cores for transport, output acceptance, evidence, retry, and auth behavior.

Agent behavior

Noema

Public repositories try three approved NVIDIA NIM hosted models first. An explicitly configured custom provider remains the final fallback. Private/internal repositories cannot use public-only trial candidates.

OpenCode Agent

The existing configured pool is intersected with the shared manifest and reordered: NVIDIA NIM free models → opencode-free/* → included-quota GitHub Models → paid OpenCode/OpenAI/OpenRouter candidates. The unchanged core still owns retries, output normalization, evidence sealing, secret masking, and fail-closed exhaustion.

Strix

The shared plan is applied before the unchanged Strix scan gate resolves its primary model. The gate still selects provider-specific keys/API bases, parses security findings, enforces severity thresholds, and fails closed on provider warning or timeout signals.

Verification

Local exact-slice verification established:

  • 74 passed
  • central Python policy surface: 487 statements, 180 branches, 100%
  • contextual_fallback_policy.py: 174 statements, 56 branches, 100%
  • Noema wrapper: 93 statements, 36 branches, 100%
  • vendored contextual-orchestrator planner/parser/types/facade: 100% statement and branch coverage
  • all newly public Python symbols have docstrings
  • compileall passed
  • Bash syntax checks passed for the OpenCode adapter and Strix model utility

A source-pin regression additionally requires the adapter and VENDOR_RECEIPT.json to identify integrated upstream commit 40c6a4b419cdf8fa90c422acb5443a0e1cca5d16. Repository-wide exact-head checks remain authoritative.

Cross-repository dependency

ContextualWisdomLab/contextual-orchestrator#96 combines the provider-egress hardening and interpreter-portable Atheris prerequisite. Stacked PR ContextualWisdomLab/contextual-orchestrator#94 adds the transport-neutral free-first policy on that integrated base, producing immutable source commit 40c6a4b419cdf8fa90c422acb5443a0e1cca5d16 pinned here.

This central PR must not merge until #96 and then #94 have passed exact-head checks and merged without bypass. The receipt verifies the integrated source commit, exact file maps, regular-file status, and unchanged fallback-policy Git blob identities, preventing mutable-branch or ancestor-only substitution.

Documentation

  • docs/shared-llm-fallback-policy.md
  • docs/doctoring/shared-llm-fallback-policy.md with APA 7 references
  • CHANGELOG.md

Verification gate

Merge only after the cross-repository dependency order completes and exact-head central tests, 100% statement/branch coverage, complete docstrings, Secret Scan, Semgrep, CodeQL, Python Security, supply-chain checks, Noema, OpenCode, Strix, CodeRabbit, independent current-head approval, and branch protection pass. No reviewer credential name or scope changes.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 2 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 398cce57-6973-42f2-8b3d-7f4ecd77e86d

📥 Commits

Reviewing files that changed from the base of the PR and between 3f65dbe and 49aacff.

📒 Files selected for processing (28)
  • CHANGELOG.md
  • config/llm-fallback-policy.json
  • docs/doctoring/shared-llm-fallback-policy.md
  • docs/shared-llm-fallback-policy.md
  • requirements-strix-ci-hashes.txt
  • requirements-strix-ci.txt
  • scripts/ci/contextual_fallback_policy.py
  • scripts/ci/noema_review_gate.py
  • scripts/ci/noema_review_gate_core.py
  • scripts/ci/run_opencode_review_model_pool.sh
  • scripts/ci/run_opencode_review_model_pool_core.sh
  • scripts/ci/strix_model_utils.sh
  • tests/conftest.py
  • tests/test_contextual_fallback_policy.py
  • tests/test_contextual_fallback_policy_repository.py
  • tests/test_contextual_fallback_policy_source_pin.py
  • tests/test_fallback_policy_semgrep_boundaries.py
  • tests/test_noema_fallback_policy.py
  • tests/test_shared_llm_fallback_adapters.py
  • tests/test_vendored_fallback_manifest.py
  • tests/test_vendored_fallback_plan.py
  • vendor/contextual-orchestrator/LICENSE
  • vendor/contextual-orchestrator/VENDOR_RECEIPT.json
  • vendor/contextual-orchestrator/contextual_orchestrator/__init__.py
  • vendor/contextual-orchestrator/contextual_orchestrator/_fallback_manifest.py
  • vendor/contextual-orchestrator/contextual_orchestrator/_fallback_plan.py
  • vendor/contextual-orchestrator/contextual_orchestrator/_fallback_types.py
  • vendor/contextual-orchestrator/contextual_orchestrator/model_fallback.py

Comment @coderabbitai help to get the list of available commands.

@seonghobae
seonghobae force-pushed the feat/unify-free-first-llm-fallbacks branch from 63b5c1d to 6c99cbc Compare August 4, 2026 23:41
@seonghobae
seonghobae changed the base branch from fix/strix-python-security-cves to main August 4, 2026 23:41

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@opencode-agent @cwl-noema-review exact-current-head review requested for 6c99cbc25bee6e94b61e7fbdad59afd9893e56f9. Please verify the contextual-orchestrator vendor receipt and blob identities, strict manifest parsing, public/private repository eligibility, credential-value non-disclosure, free-before-paid and free-to-free ordering, unchanged Noema/OpenCode/Strix reviewer-auth and transport boundaries, 100% statement/branch coverage evidence, and fail-closed behavior. Submit approval only for this exact head after required checks succeed.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

@seonghobae I will review PR #760 at exact head 6c99cbc25bee6e94b61e7fbdad59afd9893e56f9.

I will verify the requested policy, security, boundary, and evidence requirements. I will approve only if the exact-head review and required checks succeed.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 6c99cbc25bee6e94b61e7fbdad59afd9893e56f9.

  • Head SHA: 6c99cbc25bee6e94b61e7fbdad59afd9893e56f9

  • Workflow run: 30961482014

  • Workflow attempt: 1

Coverage evidence

Coverage Decision

  • Result: FAIL
  • Test evidence: not proven passing
  • Docstring evidence: not proven passing when configured
  • Failure count: 1

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (9 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (9 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (2 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (2 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["CI script (6 files)"]
  S3 --> I3["review and security gate shell path"]
  I3 --> R3["Review risk: CI script (6 files)"]
  R3 --> V3["bash -n plus Strix self-test"]
  Evidence --> S4["Test (7 files)"]
  S4 --> I4["regression suite"]
  I4 --> R4["Review risk: Test (7 files)"]
  R4 --> V4["targeted test run"]
Loading

@opencode-agent

opencode-agent Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

OpenCode Review Overview

  • Head SHA: 49aacff9e1a4907ae9eae4e64fad5881968a7053
  • Workflow run: 30971144855
  • Workflow attempt: 1
  • Gate result: REQUEST_CHANGES (approval step)

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 49aacff9e1a4907ae9eae4e64fad5881968a7053.

  • Head SHA: 49aacff9e1a4907ae9eae4e64fad5881968a7053

  • Workflow run: 30971144855

  • Workflow attempt: 1

Coverage evidence

Coverage Decision

  • Result: FAIL
  • Test evidence: not proven passing
  • Docstring evidence: not proven passing when configured
  • Failure count: 1

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (11 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (11 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (2 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (2 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["CI script (6 files)"]
  S3 --> I3["review and security gate shell path"]
  I3 --> R3["Review risk: CI script (6 files)"]
  R3 --> V3["bash -n plus Strix self-test"]
  Evidence --> S4["Test (9 files)"]
  S4 --> I4["regression suite"]
  I4 --> R4["Review risk: Test (9 files)"]
  R4 --> V4["targeted test run"]
Loading

Comment thread scripts/ci/contextual_fallback_policy.py Fixed
Comment thread scripts/ci/noema_review_gate.py Fixed
Comment thread scripts/ci/noema_review_gate.py Fixed

Copy link
Copy Markdown
Contributor Author

Exact-head security repair at 6396a8217e0e2737295fb40b4d8747b2f2e25343:

  • aligned the Strix dependency closure to the reviewed aiohttp==3.14.3 / cryptography==50.0.0 floor;
  • retained Git SHA-1 only as the mandatory Git object-identity algorithm, now with usedforsecurity=False and a rule-scoped documented suppression;
  • documented the Noema shared-globals loader as executing only the fixed regular non-symlink sibling from the immutable trusted workflow checkout, with an exact rule-scoped suppression;
  • added permanent regression contracts for both boundaries.

The previous dependency-review and Semgrep failures were therefore addressed without excluding either file, weakening the scanners, changing reviewer credentials, or changing the free-first execution policy. Fresh exact-head checks remain authoritative, and cross-repository order #96#94#760 is unchanged.

Comment thread scripts/ci/noema_review_gate.py Fixed

Copy link
Copy Markdown
Contributor Author

@opencode-agent @cwl-noema-review Please independently review exact current head 49aacff9e1a4907ae9eae4e64fad5881968a7053. Exact-head Python Security, Security Scan, CodeQL, Semgrep, Secret Scan, OSV, Scorecard, and SBOM are successful; the earlier REQUEST_CHANGES targets superseded 6c99cbc25bee6e94b61e7fbdad59afd9893e56f9, and all scanner threads are resolved/outdated. Verify the receipt-pinned contextual-orchestrator source, strict manifest parser, free-before-paid planning, reviewer transport/identity/key preservation, exact rule-scoped trusted-sibling exec exception, no secret-value serialization, and current secure Strix lock. Do not merge until contextual-orchestrator #96 and then #94 have merged and this branch has been reconciled against protected main.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 49aacff9e1a4907ae9eae4e64fad5881968a7053.

  • Head SHA: 49aacff9e1a4907ae9eae4e64fad5881968a7053

  • Workflow run: 30971144855

  • Workflow attempt: 1

Coverage evidence

Coverage Decision

  • Result: FAIL
  • Test evidence: not proven passing
  • Docstring evidence: not proven passing when configured
  • Failure count: 1

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (11 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (11 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (2 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (2 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["CI script (6 files)"]
  S3 --> I3["review and security gate shell path"]
  I3 --> R3["Review risk: CI script (6 files)"]
  R3 --> V3["bash -n plus Strix self-test"]
  Evidence --> S4["Test (9 files)"]
  S4 --> I4["regression suite"]
  I4 --> R4["Review risk: Test (9 files)"]
  R4 --> V4["targeted test run"]
Loading

@seonghobae
seonghobae marked this pull request as draft August 5, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants